Skip to content

Conversation

xrmx
Copy link
Contributor

@xrmx xrmx commented Jul 11, 2025

Description

This introduces a basic OpAMP http client for handling remote configuration. The client implements a bunch of capabilities (ReportsStatus, ReportsHeartbeat, AcceptsRemoteConfig, ReportsRemoteConfig) that are enough to get a remote config from an opamp server, parse it, apply it and ack it. Since OTel / OpAMP do not standardize APIs, config options or environment variables the distros are required to provide code doing so.
OTel Python distros would need to provide their own message handler callback that implements the actual change of whatever configuration their backends sends.

In practice distro would need to do something like the following:

from opentelemetry._opamp import messages
from opentelemetry._opamp.agent import OpAMPAgent
from opentelemetry._opamp.client import OpAMPClient
from opentelemetry._opamp.proto import opamp_pb2 as opamp_pb2


def opamp_handler(agent: OpAMPAgent, client: OpAMPClient, message: opamp_pb2.ServerToAgent):
    for config_filename, config in messages._decode_remote_config(message.remote_config):
        print("do something")


class MyOpenTelemetryConfigurator(_OTelSDKConfigurator):
    def _configure(self, **kwargs):
        super()._configure(**kwargs)

        enable_opamp = False
        endpoint = os.environ.get(OTEL_OPAMP_ENDPOINT)
        if endpoint:
            # this is not great but we don't have the calculated resource attributes around
            # see https://github.com/open-telemetry/opentelemetry-python/pull/4646 for creating
            # an entry point distros can implement
            resource = OTELResourceDetector().detect()
            agent_identifying_attributes = {
                "service.name": resource.attributes.get("service.name"),
            }
            opamp_client = OpAMPClient(
                endpoint=endpoint,
                agent_identifying_attributes=agent_identifying_attributes,
            )
            opamp_agent = OpAMPAgent(
                interval=30,
                message_handler=opamp_handler,
                client=opamp_client,
            )
            opamp_agent.start()

The module is called _opamp because it's a bit early to standardize on an api. The code is divided roughly in:

  • agent: handles threads and queues for sending messages to the server, supports sending heartbeat messages at a fixed interval
  • client: expose the api to build and send the OpAMP messages
  • messages: wrappers to simplify protobuf serialization
  • transports: http backends, currently there is only one using requests (to match the exporters)

OpAMP reference: https://opentelemetry.io/docs/specs/opamp/.

This is tested against https://github.com/elastic/opentelemetry-collector-components/ that is using the opamp-go implementation.

TODO:

  • write some docs with usage examples

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

  • tox

Does This PR Require a Core Repo Change?

  • Yes. - Link to PR:
  • No.

Checklist:

See contributing.md for styleguide, changelog guidelines, and more.

  • Followed the style guidelines of this project
  • Changelogs have been updated
  • Unit tests have been added
  • Documentation has been updated

@xrmx xrmx requested a review from a team as a code owner July 11, 2025 15:06
@danielgblanco
Copy link

@open-telemetry/opamp-spec-approvers could you help review this? thanks!

@xrmx xrmx force-pushed the basic-http-opamp-client branch from 3e67115 to e5fced7 Compare July 30, 2025 16:59
@xrmx xrmx force-pushed the basic-http-opamp-client branch from 1d2061f to 04d8923 Compare August 1, 2025 15:11
@xrmx xrmx force-pushed the basic-http-opamp-client branch from 4cfbf33 to a746292 Compare September 11, 2025 09:28
@xrmx xrmx force-pushed the basic-http-opamp-client branch from 1d85953 to cf479f2 Compare September 12, 2025 09:51
@Kludex
Copy link
Member

Kludex commented Sep 12, 2025

I can't unresolve threads, but I've replied above.

@tigrannajaryan
Copy link
Member

Thank you for working on this.

This is tested against https://github.com/elastic/opentelemetry-collector-components/ that is using the opamp-go implementation.

One thing that would be great to add is interoperability tests between Go and Python implementations. Ideally we would have both pairs (Client in Python, Server in Go and the vice versa) connecting and performing OpAMP exchanges to make sure the pair works correctly together.

OpAMP Go implementation presumably is the most complete implementation at the moment, so all other languages (including Python) could use it as a reference implementation to test against. I think we can spend a bit time adding any necessary tooling to opamp-go that makes this possible (mock servers, clients that probe capabilities, etc). Unfortunately I don't have time myself but if anyone wants to work on the design and implementation I can dedicate some time to review the design.

@xrmx
Copy link
Contributor Author

xrmx commented Sep 15, 2025

Thank you for working on this.

This is tested against https://github.com/elastic/opentelemetry-collector-components/ that is using the opamp-go implementation.

One thing that would be great to add is interoperability tests between Go and Python implementations. Ideally we would have both pairs (Client in Python, Server in Go and the vice versa) connecting and performing OpAMP exchanges to make sure the pair works correctly together.

This PR has some recorded e2e tests doing this, we don't test many scenarios but at least we test against a real response.

OpAMP Go implementation presumably is the most complete implementation at the moment, so all other languages (including Python) could use it as a reference implementation to test against. I think we can spend a bit time adding any necessary tooling to opamp-go that makes this possible (mock servers, clients that probe capabilities, etc). Unfortunately I don't have time myself but if anyone wants to work on the design and implementation I can dedicate some time to review the design.

That would be helpful indeed

@xrmx xrmx force-pushed the basic-http-opamp-client branch from b48e242 to 3eb1769 Compare September 30, 2025 08:36
Copy link
Member

@aabmass aabmass left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Giving this an approving "python review" but I don't know much about OpAmp. Looks good enough to merge though as an initial prototype with proper disclaimers.

🚢

agent.send(payload=message)

opamp_client = OpAMPClient(
endpoint="http://localhost:4320/v1/opamp",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be good to include some instructions on how to regenerate the VCR fixtures.

Alternatively, would it be possible to run the real server during the actual tests to avoid having to maintain fixtures? If it's easy enough and not flaky, it would be better IMO.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have some instructions but they are using an Elastic specific collector sending data to Elastic cloud: https://github.com/elastic/elastic-otel-python/blob/main/tests/opamp/README.md
If it's fine I can copy them otherwise I can add a TODO to recreate them with just the opamp-go server implementation.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah i see, there's no pure collector-contrib OpAmp server?

Copy link
Contributor Author

@xrmx xrmx Oct 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is an opamp-go example server but it doesn't offer a way set a config file for the agents without touching the code. Filed this open-telemetry/opamp-go#456

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants